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Introduction 


Americans love ranking systems. Whether we are ranking the hottest celebrities, the top 
ten singles, the top chef, or the next design star, ranking seems to be built into the 
American psyche as a symptom of our competitive, aspirational nature, and our desire to 
quickly understand the value of things. 


This desire has fueled a veritable ranking industry with respect to institutions of higher 
education. There is great diversity in what ranking systems purport to rank. Beyond the 
well-known U.S. News and World Report (USNWAR) “Best Colleges’ lists are those that 
rank specific institutional types, such as flagship, land-grant, and international 
universities. Others aim to provide information to consumers about how a particular 
college or university rates on anything from “value,” broadly defined, to institutional 
commitment to sustainability, gender balance within the student population, and salaries 
of recent graduates, to the perceived quality of its athletic facilities, the popularity of its 
website, or its status as a party school. 


On the whole, ranking systems rely on those aspects of higher education that are easy 
to count. For example, USNWA criteria include student selectivity as measured by SAT, 
high-school GPA or rank, faculty resources (which include class size, faculty salaries, 
faculty qualifications and appointment types), graduation and retention rates, spending 
per student, alumni giving, and graduation-rate performance. Further, dominant ranking 
systems, such as USNWR, tend to measure a diverse array of colleges and universities 
based on the characteristics of research universities. Nevertheless, the ranking industry 
has become very influential in higher education, raising questions about the relationship 
between dominant ranking systems and the purposes and goals of institutions of higher 
education. Whose interests do the most widely-used rankings systems serve, and why? 
What purposes and ideologies are they not representing? 


Questions such as these led the first author of this paper to develop a graduate course 
on ranking systems in higher education. The second author was first a student, then a 
speaker in the course. To our knowledge, this is one of a very few graduate courses on 
the topic. To date, the class has run three times, with a total of 29 students exploring 
ranking systems and their role. The course is designed to tackle the rankings issue from 
various angles, through examination of the considerable body of literature on the topic 
and the multiple perspectives of guest speakers on the benefits and drawbacks of 
ranking systems. As a capstone project, students work together to develop a new 
ranking system to emphasize institutional qualities that are often neglected in the 
popular, mainstream ranking systems. They then run a trial test of their methodology 
using publicly-available data sets. For example, students use NSEE, the Common Data 
Set, and IPEDS to rank ten institutions on issues of first-year student retention efforts, 
career development, or opportunities for student leadership. 


The purpose of this article is to present our critique of the main weaknesses and 
contributions of dominant ranking systems, to consider some of the positive and/or 
neutral roles that they are serving, and to offer three examples of purposes and goals of 
higher education we think they are not fulfilling. This critique is based on an extensive 
review of over 100 articles on ranking systems as well as notes from class discussions 
aimed at unpacking their role in higher education. 


NERCHE Working Paper: “Inside Rankings: Limitations and Possibilities” Page 3 


We began our classroom discussions by considering the criteria popular ranking 
systems such as USNWA use to rank academic programs (Appendix A outlines some of 
the more popular ranking systems) and then moved onto an examination of some new 
alternative ranking systems. Our framework for analyzing each of these systems is 
included in Appendix B. 


In the following section, we consider the weaknesses of the ranking systems we 
analyzed, as well as some of the benefits higher education and its stakeholders have 
gained via the creation of these ranking systems. 


Weaknesses of the Dominant Ranking Systems 


While ranking systems may provide a quick and useful way to sort institutions along a 
wide range of institutional characteristics, they are also highly problematic. Birnbaum 
(2006) observes that “ratings or rankings pretend to be objective and scientific; in reality, 
however, they are manifestations of ideologies about the purposes of higher education” 
(p. 9). The dominant ideology represented in these ranking systems favors the practices 
and policies of prestigious research universities, to the detriment of the many diverse 
colleges and universities that educate the lion’s share of undergraduate students. An 
especially potent example of this ideology is that “high quality” research is determined by 
where that research is published, how often it is cited, and the number and kinds of 
awards and fellowships faculty at a particular institution receive. Additionally, because it 
is difficult to measure academic quality quantitatively, more often than not research 
reputation is taken as a proxy for academic program quality, a relationship that can be 
misleading and can result in the unfair comparison of high-quality programs at lesser- 
known universities to mediocre departments at universities with greater name 
recognition (Marginson, 2006). 


Dominant ranking systems rely heavily on survey instruments that are completed by 
administrators, faculty, and students to gauge academic reputation. USNWA criteria, for 
example, include academic reputation as measured by a peer review survey of 
academics. An unintended consequence of such a practice is that raters may have a 
bias toward an institution based on its past rankings or on its elite status. For example, 
one third of those who responded to an American survey ranked Princeton among the 
top ten law schools in the country; however, despite attempts to reconstitute it in recent 
decades, Princeton’s law school hasn't existed since 1852. 


Nonetheless, the primacy of institutional reputation in dominant ranking systems persists 
and leads many consumers to conflate reputation with a quality undergraduate 
education while overlooking colleges and universities — including special-interest 
institutions that popular rankings tend to neglect (Gasman, 2010) — which might provide 
a better educational experience. 


Research- and reputation-based rankings privilege the creation and dissemination of 
scientific knowledge, in particular those fields, such as biomedicine, in which faculty 
publish articles in highly regarded journals like Science and Nature. Increasingly, U.S. 
research universities compete in international ranking systems such as the Academic 
Ranking of World Universities, Times Higher Education supplement, and Webometrics’ 
ranking of world universities. These world university ranking systems place even greater 
emphasis on measuring research performance and prestige in the hard sciences than 
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USNWAR (Hazelkorn, 2009; Marginson, 2006). International research university ranking 
systems focus even more explicitly on the percentage of faculty who hold Nobel prizes 
and field medals and bring in research-related external funding. Further, these systems 
assume a relationship between scholarship that is widely cited in scientific journal 
indexes, and quality. The “Matthew effect,” whereby the scientific reputation of the 
author influences reception of the scientific findings (Braxton, 1986), reveals the flaw in 
this assumption. Merton (1968) observed that “a scientific contribution will have greater 
visibility in the community of scientists when it is introduced by a scientist of high rank 
than when it is introduced by one who has not yet made this mark” (p. 59). Therefore, 
some faculty may have an undue impact on institutional rankings based more on their 
accumulated prestige than on current accomplishments. 


At the other end of the spectrum, a significant body of faculty work is effectively excluded 
from dominant ranking systems because the principal language of peer-reviewed 
publications is English, thereby disadvantaging non-English speaking scholars in the 
citation index. 


Below we identify several other key weaknesses associated with dominant ranking 
systems: 


Strategic Imitation. Currently, research- and reputation-based ranking systems are 
having a disproportionate effect on the vast landscape of higher education in the U.S. 
and around the world. As several national and international studies demonstrate, such 
rankings exert considerable influence over institutional decision making in critical areas, 
including faculty compensation and institutional mergers (Hazelkorn, 2009; IHEP, 2009; 
Marginson, 2006; Meredith, 2004; Winston, 2000). An example of the extent of this 
influence is the role that they play in perpetuating a phenomenon identified by Rhoades 
(2010) as “strategic imitation,” the mimicking of behaviors of higher ranked institutions, 
or “aspirational peers,” that lower-ranked institutions seek to emulate. For example, upon 
assuming their positions, new presidents and provosts often identify a set of aspirational 
peers for the purposes of strategic planning. As part of the process, the names of the 
aspirational peers then feature prominently in the everyday rhetoric of institutional 
leaders and in the benchmarking exercises of institutional researchers. The drive to rise 
in rankings provides a powerful incentive for institutional leaders to restructure academic 
programs and even institutions as a whole so that they more closely resemble 
aspirational peers at the expense of institutional and programmatic diversity, both 
regionally and nationally (Rhoades et al, 2008; Rhoades, 2010). 


Flawed Instrumentation. Too often, inexact wording in survey instruments leads to 
confusion. For example, those completing surveys may not understand what measures 
of quality refer to which unit (e.g., department versus program versus university). 


Focus on Inputs. Because dominant rankings rely heavily on inputs, simply knowing the 
input measures, such as student selectivity, age of institution, endowment, faculty 
productivity, and alumni giving, can be sufficient to predict an institution’s ranking. For 
example, Webster (2001) showed a severe and pervasive set of relationships between 
average SAT score and predicted retention and graduation rates. Webster further found 
average SAT and ACT scores of incoming students to be the most influential criteria in 
determining where an institution ranks. While these relationships reveal nothing about 
the quality of the undergraduate experience, they reinforce an unfortunate tendency 
among colleges and universities to recruit students whose SAT scores can be read as a 
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measure of institutional quality. Kuh and Pascarella (2004) arrived at a similar finding in 
their examination of 50 universities participating in USNWR rankings. 


Additionally, dominant ranking systems tend to imbue certain inputs with greater 
significance or predictive value than is warranted. For example, ranking systems 
associate the number of prolific faculty employed at an institution with the quality of the 
institutional environment. Yet, the fact that highly productive faculty tend to be 
associated with prestigious institutions may have more to do with their preference for 
working at these institutions than institutional efforts to create a supportive environment 
for faculty work. 


Limited Mobility within the Ranking System. Though ranking systems imply a certain 
level of fluidity, there is in fact remarkably little long-term mobility at the top of the 
USNWAR rankings. Johns Hopkins, for instance, has always ranked between ten and 22, 
and Harvard between one and three (Dichev, 2001). Because the institutions being 
ranked vary with respect to contexts and resources, and dominant ranking systems favor 
highly-selective and resource-rich institutions, possibilities for mobility are largely 
illusory. Even when evidence of mobility exists, research shows that it is based less ona 
change in “quality” and more on shifts in measurements and methodologies from year to 
year. Dichev (2001) found most of the changes in the rankings (70-80%) are due to 
meaningless “noise’—i.e., changes in measurement estimation and information 
processing errors in the ranking’s underlying components. “About 30 percent of a given 
change in the rankings reverses in the very next ranking and another 23 percent 
unravels in the 2 year ahead ranking” (p. 248). In one year the ranking systems may give 
more statistical weight to a measure that has attracted much interest in higher education 
(such as cash value of fringe benefits adjusted for regional variations), resulting in shifts 
in the ranking status for some institutions. Once interest has waned, however, the 
category is eliminated, and the rankings stabilize. Dichev notes changes in the 
weightings of financial resources, faculty resources, and alumni satisfaction between 
1991 and 1993. Whereas changes in fundamental school quality account for only about 
ten percent of the changes in published school rankings, variation in changes for faculty 
resources is the main driver of that ten percent. 


Questionable Rewards. One could argue that attracting more academically talented 
students and faculty skilled at drawing outside funding benefits institutions in several 
ways. First, doing so increases institutional prestige. Also, because a large percentage 
of academically talented students come from middle- and upper-middle-class 
households with parents that can afford to pay all or most of the tuition and fees, an 
institution might see financial gains associated with reduced need to offer financial aid. 
In fact, students most likely to rely on rankings come from second- or third-generation 
college-educated homes and have significant cultural capital and resources 
(McDonough et al, 1995). Indeed dominant ranking systems reward institutions that 
strive to attract students who are almost guaranteed to be academically successful and 
faculty who increase institutional resources by obtaining outside funding. But the 
administrative costs of recruiting and supporting these groups are considerable. In fact, 
rankings contribute to a winner-take-all environment: The ranking systems serve to 
reward those at the top with the most resources. The same is true within institutions, as 
departments that are highly ranked receive more resources to improve themselves; 
those with poorer rankings the least (Hazelkorn, 2011). 
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Perhaps even more disquieting is the price that society pays when colleges and 
universities make decisions that prioritize self-interest over the interests of the public 
realm. The race to enroll academically successful and financially flush students 
ultimately hampers the diversity and social mobility functions of the higher education 
system. 


Disturbing Consequences. Assessing the performance of many college presidents 
(much less directors of admissions and development) based on such factors as 
admissions and alumni giving has led to some dubious behavior. The Chronicle of 
Higher Education and the New York Times have been replete with examples of higher 
education administrators who fabricated the numbers in order to move their institution up 
in the rankings. Additionally, efforts to reach certain institutional targets incentivize 
institutions to accept more students on early-decision, to reject qualified applicants who 
are likely not to attend, and to recruit students to apply who are not likely to get in. Most 
of these same institutions require their students to sign honor codes saying they will not 
cheat, yet many are modeling exactly the opposite behavior in their attempts to game 
the dominant ranking systems to their advantage. 


Of further concern is that most of the dominant research- and reputation-based rankings 
are published for for-profit businesses whose underlying profit motive may hamper their 
ability to objectively assess quality. This may lead such companies to oversimplify 
complex criteria, cater to certain stakeholders within the market (such as MBA programs 
as opposed to TRIO students and their families), and be resistant to change criteria that 
are fatally flawed but popular. 


Most important, we argue that the dominant ranking systems miss certain essential 
purposes and goals of higher education. The research- and reputation-based rankings 
rarely address important process issues like student engagement, hours spent studying, 
and value-added college experiences, including internships, study abroad opportunities 
and service learning. They shed no light on how college shapes people as learners and 
citizens, contributes to social change, revitalizes communities, or enhances social 
mobility. We recognize that rankings could not possibly be all things to all people, but we 
think the dominant ranking systems and the attention they receive have marginalized 
many significant roles that higher education can play in the lives of individuals and in 
society at large. We will explore this argument further after discussing some of the 
assets of ranking systems in the next section. 


Benefits of Ranking Systems 


Before acknowledging the benefits of ranking systems, we want to note that there have 
been significant improvements in USNWR methodologies over the last five years, in 
ways that mitigate some of the limitations we have mentioned, such as decreasing the 
weight of reputation-based criteria, extending the reputational survey mailings to include 
admissions counselors, creation of alternative niche rankings (such as for service- 
learning programs). For example, in the case of expanding reputational surveys to 
guidance counselors, they have added a new perspective to that criterion. Guidance 
counselors work with students and their families and have a distinct perspective on what 
colleges are offering and who they are attracting. Likewise, on the international stage, 
Times Higher Education recently retooled its ranking system with the help of a new 
partner (Thomson Reuters, replacing QS) by incorporating measures intended to reflect 
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teaching quality, as well as measures for research quality, innovation, and 
internationalization. 


There are several ways in which ranking systems can benefit individual institutions and 
higher education as a whole. First, some of the criteria, especially many of the newer 
ones, provide a reason to collect data on noteworthy aspects of higher education such 
as Class size and faculty-student ratios, four-year graduation rates, students receiving 
nationally competitive awards, and four-year debt load for typical student borrowers. As 
such, the ranking systems require institutions to produce data that can be mined for 
institutional research and used for institutional improvement. Second, even though there 
is limited mobility within dominant ranking systems, occasionally a less visible institution 
benefits from the free marketing that accompanies media attention for an improved 
ranking. Third, ranking systems create an external accountability medium in the areas 
they evaluate, isolating areas of importance for institutions to improve upon in order to 
achieve upward movement in the rankings. Ranking systems can prioritize and 
incentivize certain areas of measurement that can benefit society as a whole—for 
instance, boosting graduation rates. Fourth, the world rankings have caused many more 
American research universities and faculty to pay attention to their competition in other 
countries and may even have promoted additional exchange of scholars and students. 
Fifth, institutions that have seen improvement in the rankings over short or long periods 
of time can use the rankings to boost morale or celebrate institutional progress toward a 
goal, providing a motivating force for faculty and staff. Finally, dominant ranking systems 
provide information to consumers and stakeholders that may help them navigate some 
of the complexities associated with institutions of higher education. 


Perhaps most important to us though, the dominant research- and reputation-based 
rankings have created a forum for dialogue about the purposes of higher education and 
how to measure quality. In the next section we explore recent additions to the ranking 
system industry which expand this dialogue. 


Recent Alternatives to Dominant Rankings 


As many readers know, there have been some important newcomers to the world of 
rankings. Three examples are Washington Monthly, The Voluntary System of 
Accountability, and the Education Trust’s rating of public flagships in a report entitled 
Opportunity Adrift. Washington Monthly's ranking system includes criteria for service (as 
measured by the size of Army/Navy ROTC programs relative to the size of the school, 
the number of alumni currently serving in the Peace Corps, and the percentage of 
federal work-study grant money spent on community-service projects) and criteria for 
social mobility (as measured by the percentage of Pell grant recipients, SAT scores, and 
graduation rates)—which serve as indicators of an institution’s efforts to enroll low- 
income students. As Washington Monthly observes, “our approach is fundamentally 
different from USNWAR and similar guides: instead of focusing on what colleges can do 
for you, we ask what colleges are doing for the country by improving social mobility, 
producing research and promoting public service” (Glastris, 2009). 


The Voluntary System of Accountability recommends that institutions use one of three 
measures—the Measure of Academic Proficiency and Progress, Collegiate Assessment 
of Academic Proficiency, or Collegiate Learning Assessment—to assess student 
learning. For example, the Collegiate Learning Assessment “presents realistic problems 


NERCHE Working Paper: “Inside Rankings: Limitations and Possibilities” Page 8 


that require students to analyze complex materials and determine the relevance to the 
task and credibility. Students’ written responses to the tasks are evaluated to assess 
their abilities to think critically, reason analytically, solve problems and communicate 
clearly and cogently” (Council for Aid to Education, 2009). Shulenburger, Mehaffy, and 
Keller (2008) point out that such measures provide an alternative to standardized tests 
that cannot measure high-level skills. Students, however, take the assessments on a 
voluntary basis, so there are concerns about the representativeness of the sample and 
the degree to which the three instruments are measuring the same skills. 


In early 2010, the Education Trust released a report and ranking, Opportunity Adrift, a 
follow-up to Engines of Inequality, an unprecedented 2006 study of access and success 
with respect to 50 U.S. public flagship universities—typically the oldest, largest, and 
most prestigious in each state. Opportunity Adrift includes ratings on access for low- 
income and underrepresented minority students, on the relative success of students 
from those groups in earning a degree, and on the changes in these ratings over time. 


After having examined the existing world of ranking systems, our class deliberated on 
the areas where we felt dominant ranking systems and newer alternatives were paying 
insufficient attention. What kinds of higher education contributions and outcomes were 
left out of the spotlight? What kinds of unintended consequences of ranking systems 
need to be avoided? Our deliberations were informed by the views of several class 
speakers whose professional work intersects with the issue of rankings in higher 
education (i.e., George Mehaffy from AASCU, Peter Eckel from ACE, Alan Bloomgarden 
from Mount Holyoke, Sean Simone from NCES, Mona Levine and Mike Pascarella 
George from UMCP, Bob Morse from USNWA, and Ellen Hazelkorn and Brian Sponsler 
from IHEP). Finally, we considered the priorities and values we ourselves bring to the 
higher education enterprise—what we think is important and needs to be monitored, 
assessed, and celebrated. In the next section, we outline three areas that emerged from 
class discussions and that we see as emblematic of critical purposes of higher education 
that, unranked, could suffer from a lack of institutional prioritization. 


Ranking Alternative, but Core, Commitments of Higher Education 


We argue that the dominant research- and reputation-based ranking systems obscure 
the innovative and change-oriented goals of colleges and universities that are both vital 
to the well-being of the communities in which they reside and that educate the vast 
majority of an increasingly diverse student population. The first author on this paper, for 
example, has studied faculty community engagement for many years, as well as 
academic reward systems. In her research—whether exploring the use of a broader 
definition of scholarship in promotion and tenure decisions, or the challenges that 
community-engaged faculty face in the academy—faculty and administrators repeatedly 
referred to USNWA as a major obstacle to change in higher education. (To be fair, they 
could also have targeted The Princeton Review or Shanghai Jiao Tong’s Academic 
Ranking of World Universities.) The issue raised by these faculty and administrators is 
that the criteria for and measurement of excellence included in the most dominant 
national and international ranking systems favor very traditional measures of research 
productivity. As referenced earlier, such measures include publication in peer-reviewed 
journals, national research awards, and the procurement of external grant dollars. Often 
community-engaged scholars disseminate their scholarship in more applied venues such 
as policy reports, grant applications, and curriculum guides that are not recognized in 
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rankings criteria. Further, rankings devalue interdisciplinary work by emphasizing 
citations in traditional discipline-based journals and failing to collect data on joint 
appointments as well as on interdepartmental collaboration. Omissions such as these 
effectively ignore a growing awareness within the academy that disciplinary approaches 
alone are insufficient to understanding and solving the problems of a complex world as 
well as the central role that many colleges and universities play in their communities. 


Being a steward of place. As mentioned above, a leading voice in the engagement 
movement, George Mehaffy of the American Association of State Colleges and 
Universities (AASCU), served as a guest speaker in our rankings course. As an 
introduction to AASCU’s work on the concept of institutions serving as “stewards of 
place,” George posed a question we know he has asked elsewhere: “Consider two 
institutions—Eastern New Mexico and Harvard University. Now consider that an asteroid 
will unfortunately be falling on one or the other of these institutions and leave a great 
gaping hole in the ground where it once was. Which institution will be more greatly 
missed—not by the world or even the disciplines—but by the people who live all around 
that hole?” 


This colorful example aptly describes what AASCU considers a signature feature of its 
institutions—that public colleges and universities matter to the communities and regions 
that surround them. They are integrated in such fundamental ways that if they were to 
disappear, it would irrevocably harm the educational, cultural, economic, health care, 
and leadership enterprises of those communities. 


Among the 430 colleges and universities that comprise AASCU’s membership, there are 
excellent models of campuses that are deeply engaged with their communities. They 
have become the “anchor institutions” that, as Nancy Cantor (2009) notes, so many 
communities rely on for support when times are bad and as engines of growth when they 
are good. Yet, scant attention has been paid to best practices of community-engaged 
institutions by dominant ranking systems, thereby limiting opportunities to influence the 
practices of less-engaged peers. Furthermore, while community engagement leaders 
have made significant progress in benchmarking the degree to which institutions are 
engaged (see, for example, the Carnegie Classification for Community Engagement: 
http://www.carnegiefoundation.org/files/elibrary/Driscoll.pdf), little work has illuminated 
actual and specific contributions to a community and broader region. In a recent article in 
Public Purpose (Meekins & O’Meara, 2011), we suggest a set of criteria for measuring 
and ranking such contributions that institutions make as “stewards of place.” 


The kinds of contributions colleges and universities make to their communities depends 
on geography, resources, academic programs, and the condition of the communities 
themselves, making comparisons among institutions difficult. Additionally, 
institutionalizing community engagement will be influenced by factors idiosyncratic to 
individual institutional cultures and resources. The Carnegie classification, however, has 
found ways to take this into account, by evaluating commitment to community 
engagement based on the missions and resources of each individual campus rather 
than against a one-size-fits-all rubric. If a major goal of ranking systems could be to bring 
attention to important but typically under-examined institutional qualities, we believe that 
an institution’s community engagement is one such criterion worthy of inclusion. 


Developing citizens and leaders. While some important new sources of data on 
student engagement have emerged over the last 15 years (e.g., NSSE, SESSE, CIRP), 
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as well as some important new studies of student leadership, we do not have a 
systematic way of ranking or rating institutions on the degree to which they form and 
shape the civic and leadership capacity of students. Pope’s classic Colleges that 
Change Lives (2006) and Daloz et al.’s Common Fire (1996) are research studies that 
consider the characteristics of colleges that matter in cultivating individuals who serve 
the public good. Additionally, The Princeton Review now publishes a guide entitled 
Colleges with a Conscience that highlights colleges and universities that prioritize 
community involvement. We are now at a time that the American Commonwealth 
Partnership, AACU, and the U.S. Department of Higher Education have referred to as “a 
crucible moment” in democratic education. Higher education can, and in some 
circumstances does, play a critical role in enhancing the civic knowledge, skills, and 
values of students. More needs to be done, however, to showcase this critical aspect of 
higher education service. If the purpose of a college education is at least in part to 
develop leaders and civic agents involved in the political and social lives of their 
communities—and we believe it is—then those institutions that prioritize this kind of 
education and experience should be given greater visibility. 


Enhancing social mobility. |f we as a nation truly endorse the axiom that every 
American should have an opportunity to improve his or her life and standing in society 
through hard work and education, then it follows that we should reward higher education 
institutions that successfully recruit, retain, and graduate first-generation students. 
Washington Monthly has taken positive steps by including the number of student Pell- 
grant recipients among their criteria. The Gates and Lumina Foundations have made 
access to a two- and four-year degree a focus of their work and have supported 
institutions that make graduation of first-generation students a priority. AS we have 
argued above, while many existing ranking systems give points to institutions for high 
graduation rates, the process used can end up rewarding institutions that endeavor to 
recruit and admit academically talented students. The focus on improving rankings can 
obscure institutional efforts to improve the undergraduate educational environment in 
ways that facilitate the success of all students, in particular those who are the first in 
their families to attend college. Among the exemplary institutions profiled in Pope (2006) 
are many with small endowments and few academic “super stars” represented in 
incoming freshmen classes that have achieved this end. We would like to see more 
ranking criteria that reward institutions that invest in student success and enhance their 
chances for social mobility and talent development. 


We Need New Visions of the Possible 


Dominant research- and reputation-based ranking systems for undergraduate programs 
gained prominence in the 1980s and have been a major topic of study and debate ever 
since. Whether we like them or not, they are constantly influencing institutional identity, 
aspirations, and cultures. By and large, as we have argued, the assumptions and 
expectations informing dominant ranking systems are flawed by methodologies that 
mistake inputs for outcomes, a reliance on traditional forms of research and teaching 
that have limited utility in complex, dynamic, regional, national and global contexts, and 
a tendency to give short shrift to the rich diversity of institutional tyoes, missions, and 
goals that characterize the landscape of higher education. Returning to Birnbaum’s 
(2006) observations, these dominant ranking systems gain legitimacy by purporting to 
put forward objective assessments of institutional quality. However, there are ideologies 
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behind each and every criterion used about the nature of quality in higher education. 
Those who use the ranking systems often assume a high overall ranking means quality 
in areas not ranked. 


Within the academy, we contend that dominant ranking systems fail to capture central 
goals and purposes of higher education that many of us—from faculty to deans of 
students—value and that underpin our work to create educational environments in which 
all students succeed. The good news is that ranking systems continue to evolve. We 
would suggest that the seeds of ranking systems that inspire competition by rewarding 
institutional behaviors that contribute to the public good have been planted, as 
evidenced by recent changes to existing ranking systems and the addition of newer 
alternatives. Some will likely argue that evaluating institutions on measures that are not 
easily quantified, such as student leadership development, social mobility, or community 
impact, is unachievable. Admittedly, assessing things that cannot be easily counted is a 
challenge. What we assess and rank, however, tends to influence what matters in higher 
education. It is therefore important to assess the things that count. 
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Key Resources 
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(Ed.), Higher education: Handbook of theory and research: Vol. XXII (pp.121- 
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Appendices 


Appendix A: Ranking Systems for Higher Education 


Domestic Focus (examples of popular rankings) 


Publisher: 


Focus on: 


Web Reference: 


U.S. News & World Report 
(USNWR) 


Reputational surveys, 
selectivity, graduation and 
retention rates, faculty 
resources, financial resources, 
alumni giving, graduation rate 
progress 


http://colleges.usnews.rankingsand 
reviews.com/best-colleges 


The Princeton Review 


“Best 376 Colleges” 


http://www.princetonreview.com/ 
college-rankings.aspx 


Washington Monthly 


Social mobility, research, and 
service 


http://www.washingtonmonthly.com/ 
college guide/rankings 2010/ 
national university rank.php 


Forbes “America’s Best 
Colleges” 


Student satisfaction, 
postgraduate success, student 
debt, four-year graduation 
rate, competitive awards 


http://www. forbes.com/2010/08/11/ 
best-colleges-universities-rating- 


ranking-opinions-best-colleges- 
10 _land.html| 


Payscale College Salary 
Report 


Salary data from Payscale 
users 


http://www.payscale.com/best-colleges 


State University 


A mix of non-reputational, 
government reported data 


http://www.stateuniversity.com/ 


College Prowler Rankings 


A variety including campus 
dining, housing, strictness, 
social life, safety, parking, and 
weather 


http://collegeprowler.com/rankings/ 


The Chronicle of Higher 
Education’s “Great 
Colleges to Work For” 


Workplace issues including 
governance, compensation, 
benefits, career development, 
an satisfaction 


http://chroniclegreatcolleges.com/ 


“Rugg’s Recommendations 
on Colleges” 


Academic departments and 
programs 


http://www.ruggsrecommendations. 
com/ 


Bloomberg BusinessWeek 
“Best B-Schools” 


Focus on business schools: 
academic quality, student 
satisfaction, job placement 


http://www. businessweek.com/ 
business-schools/ 
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Domestic Focus (examples of popular rankings) 


Publisher: 


Focus on: 


Web Reference: 


The Center for Measuring 
University Performance 


Total research dollars, funding, 
endowments, annual giving, 
faculty awards, student 
competitiveness 


http://mup.asu.edu/ 


The Faculty Scholarly 
Productivity Index 


Citations, publications, 
research funding 


http://chronicle.com/stats/productivity/ 


Kiplinger's 100 Best Values 


Academic quality (selectivity), 
cost, and financial aid for 
public institutions 


http://www.kiplinger.com/tools/ 
colleges/ 


Princeton Review/USA 
Today, Top 100 Best Value 
Colleges 


“High-quality academics at a 
reasonable price” 


http://www.usatoday.com/news/ 
education/best-value-colleges.htm 


My Chances College 
Rankings 


Aggregated from student 
admissions decisions 


http://college.mychances.net/college- 
rankings.php 


The Global Language 
Monitor’s College Rankings 
(TrendTopper MediaBuzz 
Rankings) 


Based upon number of 
keyword appearances on the 
Internet 


http://www.lanquagemonitor.com/ 
college-rankings/ 


What Will They Learn? 
(American Council of 
Trustees and Alumni) 


Core requirements: 
composition, literature, foreign 
language, U.S. history, 
economics, math, science 


http://www. whatwilltheylearn.com/ 


International Focus (examples of popular rankings) 


Publisher: 


Focus on: 


Web Reference: 


“Academic Ranking of 
World Universities” 
(Shanghai Jiao Tong 
University) — often referred 
to as “the Shanghai 
rankings” 


Nobel and Field winners, 
citation indices, publications in 
Nature and Science, per capita 
performance 


http://www.arwu.org/ 


Times Higher Education 
(UK) (as of 2010 publishing 
separate from QS) 


Teaching, citations, research 
(volume, income, and 
reputation), international mix, 
and industry income 


http://www.timeshighereducation.co.uk 
/world-university-rankings/index.htm| 
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International Focus (examples of popular rankings) 


Publisher: 


Focus on: 


Web Reference: 


QS World University 
Rankings (UK) 
(Quacquarelli Symonds 
Limited) 


Academic reputation (peer 
review), employer reputation, 
student-to-faculty ratio, 
citations per faculty, and 
international students and 
faculty. 


http://www.topuniversities.com/ 


G-Factor International 
University Ranking (USA) 


Using Google search engine, 
ranks links to a particular 
institution from the websites of 
others 


http://universitymetrics.com/q-factor 


Webometrics World 
University Rankings on the 
Web (Spain) 


Web publication: visibility 
(external links), size, rich files, 
Google Scholar 


http://www.webometrics.info 


Maclean’s (Canada) 


Student award winners, 
student-to-faculty ratio, faculty 
grants and awards, resources, 
student support, library, and 
reputation (peer review) 


http://oncampus.macleans.ca/ 
education/rankings/ 


“International Professional 
Classification of Higher 
Education Institutions” - 
Ecole des Mines de Paris 
(France) 


Alma maters of Fortune 500 
CEOs 


http://www.mines-paristech.fr/ 
Actualites/PR/ 


Higher Education 
Evaluation & Accreditation 
Council of Taiwan 


Scientific papers citations: 
research productivity, research 
impact, and research 
excellence 


http://ranking.heeact.edu.tw/ 
en-us/2009/Page/Methodology 


RatER (Rating of 
Educational Resources) 
(Russia) 


Educational activity, research 
activity, faculty professional 
competence, financial 
maintenance, international 
activity, web volume 


http://www. globaluniversitiesranking. 
org/ 


CHE Excellence Ranking 
(Center for Higher 
Education Development) 
(Germany) 


European graduate study (by 
discipline/department) 


http://www.excellenceranking.org/ 
eusid/EUSID 


4 International Colleges & 
Universities Web Ranking 
(Australia) 


Web metrics: Google, Yahoo!, 
and Alexa 


http://www.4icu.org/ 
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International Focus (examples of popular rankings) 


Publisher: 


Focus on: 


Web Reference: 


High Impact Universities 
(Australia) 


Research Performance Index: 
quality and consistency of 
publication 


http://www. highimpactuniversities.com/ 


Scimago Institutions 
Rankings (lbero-American 
Rankings) (Spain) 


Research: scientific output, 
international collaboration, 
average scientific quality, 
publication rate 


http://www.scimagoir.com/ 
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Appendix B: Framework for Analysis of Ranking Systems 


Area of Analysis: Framing Questions: 

History Why was this ranking system created? By whom? 

Stakeholders Who is the ranking intended to serve? 

The field Who is included and excluded? What are the 
boundaries (e.g., national or international, 2- or 4-year 
institutions ?) 

Criteria What counts in this ranking system and why? 

Measurement How are the criteria evaluated? What are the methods 


for collecting data? What is the process? 


Consequences What behavior and outcomes do the rankings 
encourage? 
Key contributions and alternatives What is most distinctive and useful from this ranking? 


How could it be more effective at accomplishing its 
stated goals? 
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